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SYSTEMS AND METHODS FOR PROCESSING A DIGITAL IMAGE 

FIELD OF THE INVENTION 

[0001] The present invention is, in general, related to computer graphics and, more 
particularly, to systems and methods for processing a digital image. 

BACKGROUND OF THE INVENTION 

[0002] Tone reproduction is an important issue in digital image enhancement. Tone 
reproduction relates to the representation of brightness and darkness in a digital image and 
the relationship of the representation to the original image. Tone reproduction is associated 
with several difficulties. Brightness and darkness are not always accurately represented in a 
captured digital image. Imaging circuitry utilized to capture the digital image may have 
several limitations. For example, the imaging circuitry may inject noise into the digital 
information or may present other non-linearities in the image capturing process. 

[0003] Accordingly, techniques have been developed to improve the quality of tone 
reproduction. Two types of approaches are currently utilized: global mapping and local 
adaptive tone correction. Global mapping is a relatively straight- forward algorithm. 
However, its effectiveness is limited when applied to digital images that possess a high 
dynamic range. Local adaptive methods are generally utilized to enhance the tone 
reproduction of digital images that exhibit high dynamic ranges. Local adaptive algorithms 
seek to change a scaling factor based on local image features. For example, Chie et al. 
proposed a non-uniform scaling function for rendering high dynamic range computer 
graphics in "Spatially non-uniform scaling functions for high contrasts images," Proceedings 
of Graphics Interface y 93, 1993. Moroney disclosed an image mask based tone correction 
algorithm for digital photographs in "Local color correction using non-linear masking," IS&T 
SID 8th Color Imaging Conference, 2000. Although these algorithms are based on research 
of human eye behaviors, these algorithms produce noticeable gradient reversals known as a 
"halo effect." 

[0004] According to Moroney, low-pass filtering of an image is useful to avoid 
excessively reducing the image contrast. This is true in smooth image regions and in regions 
with slowly changing brightness. However, at high grayscale gradient regions, the blurred 
image mask produces a partially reversed band along the edges. This band becomes 
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noticeable when one or both sides of the edge are large and smooth regions and, hence, gives 
rise to the halo effect. 

[0005] To address the halo effect, a histogram adjustment algorithm based on 
human contrast sensitivity was proposed by Larson et al. in "A visibility matching tone 
reproduction operator for high dynamic range scenes " IEEE Transactions on Visualization 
and Computer Graphics, vol. 3, no. 4, 1997. This algorithm overcomes the halo effect 
problem successfully, but it assumes that the actual brightness at each pixel location is 
known. This assumption is true for computer-created graphical images. However, this 
assumption is not true for digital images captured via digital photography. Accordingly, this 
algorithm is useful for only a limited subset of digital images. 



BRIEF SUMMARY OF THE INVENTION 

[0006] In one embodiment, the present invention is related to a method for 
processing a digital image, wherein the digital image comprises at least a gray-scale 
component. The method may comprise: morphologically filtering the grayscale component 
to produce a segmentation result; low-pass filtering the gray-scale component under control 
of at least the segmentation result to produce an image mask; and enhancing tone 
reproduction of the digital image utilizing at least the image mask. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0007] FIGURE 1 illustrates processing of data structures according to 
embodiments of the present invention. 

[0008] FIGURE 2 is a flowchart associated with creation of an image mask 
according to embodiments of the present invention. 

[0009] FIGURE 3 depicts a weight curve according to embodiments of the present 
invention. 

[0010] FIGURE 4 depicts an exemplary system that may be used to implement 
embodiments of the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0011] Referring now to the drawings, FIGURE 1 depicts flowchart 100 that 
illustrates processing of data structures according to embodiments of the present invention. 
Input image 101 is received from, for example, a digital camera. Input image 101 may be a 
grayscale image or a chromatic image. Any number of encoding formats may be utilized to 
represent input image including, but not limited to, red/green/blue (RGB), Joint Photographic 
Experts Group (JPEG), graphics interchange format (GIF), Portable Network Graphics 
(PNG), and/or the like. Embodiments of the present invention are described as processing an 
image that is based upon the CIELAB color space which was developed by Commission 
Internationale de TEclairage (CIE). The tone reproduction enhancement is applied to the "L" 
component of the CIELAB image data structure while the "a*" and "b*" components may 
remain unmodified. However, it shall be appreciated that the present invention is not limited 
to any particular digital image representation or protocol. 

[0012] If input image 101 is a chromatic image, input image 101 is separated into 
grayscale component 102 and chromatic component 103. Grayscale component 102 is 
advantageously quantized and morphologically filtered to produce image mask 105 as will be 
discussed in greater detail below. Seed value 104 is calculated from grayscale component 
102 and chromatic component 103. Seed value 104 is used to control the global strength of 
the tone correction. 

[0013] Output grayscale component 106 and chromatic component 103 are utilized 
to produce output image 107. Output image 107 may be encoded utilizing any number of 
suitable protocols for storage, communication, or further processing as desired for a particular 
application. 

[0014] In accordance with embodiments of the present invention, seed value 104 
may be selected utilizing automatic and/or manual methods. For example, a scroll bar at the 
bottom of a user interface may be utilize to allow the user to increase or decrease the global 
strength of the tone correction. Additionally or alternatively, a face detection algorithm may 
be utilized to detect an appropriate region of interest (ROI) within the image (see, for 
example, Henry A. Rowley, Shumeet Baluja, and Takeo Kanade, "Neural Network-Based 
Face Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, 
no. 1, pages 23-38, January 1998), which is incorporated herein by reference. The contrast 
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between the ROI and the rest of the image may be measured and used as the indicator for the 
desired strength of the tone correction. Specifically, if the measured contrast is high, seed 
value 104 should be set to a relatively high value. 

[0015] The design of image mask 105 attempts to balance two goals according to 
embodiments of the present invention. First, the design may advantageously be such that 
local image features are removed from the image mask to avoid reducing the image features 
in later processing. Second, salient image features, such as strong edges across large image 
regions, may be advantageously maintained to avoid the creation of the halo effect. In 
accordance with embodiments of the present invention, morphological filtering is utilized to 
implement these two goals. 

[0016] FIGURE 2 depicts exemplary flowchart 200 that illustrates the creation of 
image mask 105 according to embodiments of the present invention. In step 201, grayscale 
component 102 is received. In step 202, quantization of grayscale component 102 may 
occur. Quantization advantageously reduces the number of gray levels of the original image 
to "n-levels." Quantization reduces the complexity of the filtering operations which will be 
discussed below. In accordance with embodiments of the present invention, two hundred 
fifty-six (256) gray levels may be present in grayscale component 102. Quantization may 
reduce the number of gray levels to six (6) gray levels (i.e., n=6). The quantization may 
utilize any suitable quantization algorithm. For example, the Lloyd-MAX algorithm may be 
employed utilizing a histogram representation of the original gray levels. 

[0017] In step 203, the quantized grayscale component is decomposed into binary 
images. A binary image may created on the basis of points in the quantized gray scale image 
that are greater than a grayscale threshold. Specifically, let 7(x,y) denote the quantized 
grayscale image and t,eR (where ti is preferably one of the quantization levels) denote the 
grayscale threshold for a respective binary image, the respective binary image (Bj(x,y)) may 
be defined as: 



[0018] In step 204, the binary images are morphologically filtered. Morphological 
filtering is based on the theoretic foundation of set theory and the mathematical theory of 
order as is described in greater detail in, for example, "Mathematical Morphology: A Modern 
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Approach in Image Processing Based on Algebra and Geometry/' SIAM Review, Vol. 37, No. 
1, pp. 1-36 (March 1995) by Henk J. A. M. Heijmans, which is incorporated herein by 
reference. Morphological filtering applies filtering operations to a set (e.g., a set of pixels) 
with a structuring element. Morphological filters are non-linear and are operable to increase 
the subjective quality of digital images. A dilation operation and an erosion operation are the 
two basic operations of morphological filtering. Qualitatively, a dilation operation is 
operable to cause enlargement of "dark" objects and an erosion operation is operable to cause 
enlargement of "light" objects. 

[0019] Opening and closing operations are two other operations frequently utilized 
in morphological filtering. An open operation is an erosion operation followed by a dilation 
operation. A close operation is a dilation operation followed by an erosion operation. It shall 
be appreciated that a dilation operation followed by an erosion operation does not produce the 
original image, because the operations are non-linear. Likewise, an erosion operation 
followed by a dilation operation does not produce the original image for the same reason. 
Instead, the first operation tends to predominate due to the non-linearity of the operations. 

[0020] In accordance with embodiments of the present invention, open and close 
operations are used to perform the morphological filtering. A binary image Bi(x,y) contains a 
foreground set F (e.g., the "ones" of the binary image) and a background set F c (e.g., the 
"zeros" of the binary image) where F c is the complement of F. For the foreground set, the 
area open operation is denoted by "F ° s". The area open operation removes all connected 
regions with area less than s (e.g., an eight-connection criterion may be utilized). The area 
close operation "F • s" is the area open operation on the complementary set (i.e. F • s equals 
F c o S ). 

[0021] The morphological filtering is applied to each binary image Bi(x,y). That is, 
an open operation may be applied to each binary image followed by a close operation applied 
to each binary image. Other filtering operations may be additionally or alternatively applied. 
For example, a close operation may be applied to each binary image followed by an open 
operation applied to each binary image. The closing and opening operations are operable to 
adjust the geometrical contours of objects in the digital image. The filtered binary images 
may be represented as follows: 

B / 1 (x,y) = (B i (x,y)o S )*s 
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[0022] In step 205, the filtered binary images B 'i(x,y) are recombined to produce 
segmentation result (R(x,y)). Segmentation result (R(x,y)) may be formulated as: 



[0023] Accordingly, after quantization and morphological filtering, the image is 
simplified while retaining salient features across large regions of the image. From another 
perspective, a segmentation of the original image is created. Specifically, small or localized 
image features are removed while salient image features across large regions are retained. In 
operation, this segmentation tends to create "pseudo" boundaries at image regions with 
gradually changing grayscales due to the grayscale quantization. To address this issue, 
instead of using segmentation result (R(x,y)) directly as the image mask, segmentation result 
(R(x,y)) is used to control a low-pass filtering operation to produce image mask 105 (step 
206). In this manner, the low-pass filtering advantageously causes the filtering to occur 
across regions with lower contrast while avoiding filtering across regions with higher 
contrast. In this manner, low-pass filtering provides tone reproduction enhancement without 
creation of the halo effect. 

[0024] To describe the low-pass filtering operation in greater detail, it is appropriate 
to reiterate several terms and to define several new terms. As previously noted, I(x,y) 
represents grayscale image 102 and R(x,y) represents the segmentation result. Let w(x,y) 
represent a two-dimensional low-pass filter kernel. N(x,y) denotes the support of the low- 
pass filter (i.e. it defines the area over which the low-pass filter will operate for a particular 
pixel (x,y)). For each pixel (x 0 ,y 0 ), its neighborhood may be classified into two groups: a 
peer group defined as: 

N P (x 0 ,y 0 ) = { (x,y) | R(x,y) = R(x 0 ,y 0 ) and R(x,y) e N(x 0 ,y 0 ) } 
and a non-peer group defined as: 

N n (x 0 ,yo) = { (x,y) | R(x,y) * R(x 0 ,y 0 ) and R(x,y) e N(x 0 ,y 0 ) } 
[0025] The average grayscale value is calculated over these groups respectively as: 



R(x,y)= max{t i :B',(x,y) = l} 



I n (x,y) = 



i P (x,y) = 



Z (x ,y )e N p r ( x > y> w ( x - x o >y - y 0 ) 
Z (x , y)eNp w ( x - x o^-^o) 

Z (x , y) eN„ w(X-X 0 , J -^ 0 ) 
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[0026] By using the segmentation result (R(x,y)), embodiments of the present 
invention are operable to control the low-pass filtering upon the basis of which region a pixel 
(x,y) belongs. Specifically, if R(xi,yi) = R(x 2? y2), then (xi,y0 and (x 2? y2) belong to the same 
region. 

[0027] The contribution of the non-peer group I n (x,y) to the filtering result is then 
determined by: 

diff=|I p (x 5 y)-I n (x ? y)|/d ? 
wherein d is a value utilized to quantize the contrast (e.g., d=30 in embodiments of the 
present invention). Moreover, a weighting function (designated as a ) may be defined as 
follows: 

a = weight (diff) , 

where the weighting function may be implemented utilizing a look-up table (LUT) as 
depicted in FIGURE 3 in accordance with embodiments of the present invention. The values 
of the LUT may be tuned on an empirical basis. 



[0028] Then, image mask 105 (designated as M(x,y)) is given by: 
M(x,y) = - 



_(I p «w p »(l-QQ + I n -w n -aQ 



(w p -(l-a) + w n -a) 
where w p = I^ Np W(x-x 0 ,y-y 0 ) and w n =X (x , y)eNn w (x - x 0 ,y - y 0 ) . 

[0029] In accordance with embodiments of the present invention, image mask 105 
(M(x,y)) and seed value 104 (S) may be utilized to perform tone reproduction enhancement 
of grayscale component 102 (I(x,y)) as follows: 



l(x,y) Y 

255 J 



M(x,y)-128 
[S *28 ] 



0(x,y) = 255 

where 0(x,y) is output grayscale component 106. 



[0030] Output grayscale component 106 is provided enhanced tone reproduction. 
Output grayscale component 106 exhibits relatively smooth variation of tone between 
relatively large regions. Accordingly, the perceived quality of output grayscale component 
106 is appreciably increased. However, the control of the low-pass filtering by the 
segmentation result prevents the creation of the halo effect thereby enhancing the quality of 
the processed image. 
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[0031] FIGURE 4 depicts a block diagram of exemplary computer system 400 to 
implement embodiments of the present invention. System 400 includes display 401 to 
present graphical information to a user. System 400 further includes central processing unit 
(CPU) 402 to execute software instructions. CPU 402 may be any general purpose CPU and 
the present invention is not restricted by the architecture of CPU 402 as long as CPU 402 
supports the inventive operations as described herein. 

[0032] Computer system 400 also includes random access memory (RAM) 403 , 
which may be SRAM, DRAM, SDRAM, or the like. Computer system 400 includes ROM 
404 which may be PROM, EPROM, EEPROM, or the like. RAM 403 and ROM 404 hold 
user and system data and programs as is well known in the art. 

[0033] When implemented via executable instructions, various elements of the 
present invention are in essence the code defining the operations of such various elements. 
The executable instructions or code may be obtained from a readable medium (e.g., hard 
drive media, optical media, PROM, EPROM, EEPROM, tape media, cartridge media, and/or 
the like) or communicated via a data signal from a communication medium (e.g., the 
Internet). In fact, readable media can include any medium that can store or transfer 
information. 

[0034] For example, embodiments of the present invention store executable 
instructions or code to define tone reproduction algorithm 406 in accordance with 
embodiments of the present invention. For example, CPU 402 under the control of the 
executable instructions may process grayscale component 102 utilizing seed value 104 and 
image mask 105 to produce output grayscale component 106. 

[0035] Furthermore, computer system 400 comprises communications adapter 407 
which is adapted to couple computer system 400 to a network 408, which may be one or 
more of a telephone network, local (LAN) and/or wide-area (WAN) network, Ethernet 
network, and/or Internet network. Computer system 400 may communicate a tone 
reproduction enhanced image to another system according to embodiments of the present 
invention. 

[0036] It shall be appreciated that embodiments of the present invention are not 
limited to any particular system architecture. Embodiments of the present invention may be 
implemented utilizing any suitable processor-based or logic-based device that is capable of 
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processing a digital image. For example and without limitation, embodiments of the present 
invention may be implemented in a digital camera, personal data assistant (PDA), or any 
other processing device. 
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